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ABSTRACT 

The problem of knowing who knows what is multi- 
faceted. Knowledge and expertise lie on a spectrum and 
one's expertise in one topic area may have little bearing 
on one's knowledge in a disparate topic area. In addi- 
tion, we continue to learn new things over time. Each of 
us see but a sliver of our acquaintances' and co-workers' 
areas of expertise. By making explicit and visible many 
individual perceptions of cognitive authority, this work 
shows that a group can know what its members know 
about in a relatively efficient and inexpensive manner. 

INTRODUCTION 

Cognitive authority is the foil to administrative author- 
ity (Wilson 1983). Administrative authority is that 
which one has through rank or position. Cognitive au- 
thority is that which is granted to you by others because 
of what they think you know about. 

Cognitive authority is a subjective measurement and 
should be respected as such. There are no right answers 
to questions of cognitive authority, although, when taken 
collectively, an assessment of it can be seen as a barom- 
eter of one's standing among peers. 

Making a collective assessment visible, bringing the tacit 
individual knowledge into the realm of the explicit, and 
performing a sanity check on that assessment is the 
thrust of this paper. This work shows that a group's 
evaluations of an individual's areas of expertise can be 
gathered and potentially serve as useful loose credentials; 
loose credentials that may be useful when more expen- 
sive or heavyweight reputation cues may not be viable. 

BACKGROUND 

We satisfice; we satisfy with what is sufficient (Simon 
1957). We use what information we have to make deci- 
sions that we deem to be good enough at the time. We 
often seek out more information before making a decision 
but we have, what Simon called, "bounded rationality." 
We have imperfect information, limited attention and 
money, limited processing power and limited time, but 
we still need to make decisions. 

Choo's Decision Behavior Model shows us that contex- 
tualizcd decision making happens within organizations 
based on cognitive limits, information quality and avail- 
ability, and the values of the organization (Choo 1996). 



These inputs are handled with bounded rationality 
and within the confines of performance concerns, and 
whether the decision is good enough, among other sim- 
plifications. This decision making behavior is both ra- 
tionally expected and observed. 

Even knowing we will never have perfect information 
when working in these limited environments, we can ar- 
guably make better decisions if we can improve or in- 
crease the amount of information on hand when making 
decisions. Having more good information reduces un- 
certainty about the environment surrounding a decision, 
but it does not necessarily reduce equivocality. To re- 
duce equivocality, or ambiguity, of the information we 
have on hand, we need sensemaking and a perspective 
that comes from "retrospective interpretations" of earlier 
data and decisions (Choo 1996). We need to have seen 
this before and know what it means. What we need to 
make good decisions, in addition to good information, is 
called expertise. 

There is a vast amount of latent, untapped information 
in the environment around us. Some of it is in the built 
world, some of it is in the natural world (too big, too 
small, hidden in non-visible wavelengths, etc.), and some 
of it is in the heads of those around us. Cross noted that 
85% of managers immediately mentioned specific people 
when asked "to describe sources of information impor- 
tant to successful completion of their project" (Cross & 
Sproull 2004). They went on to write: 

As one manager said, "I mean the whole game is just 
being the person that can get the client what they 
need with [the Firm's] resources behind you. This 
almost always seems to mean knowing who knows 
what and figuring out a way to bring them to your 
client's issue" (R6). Very few of the named people 
were simply organizationally designated "experts"; 
most were described as partners in information re- 
lationships. 

If we are informed by the right people before making de- 
cisions, and they help us decide what we are looking for 
(Bclkin, Oddy & Brooks 1982), then we may improve our 
knowledge and understanding of a situation or problem 
at the time when we need to decide. Knowing from 
whom we should get our information, when we 
are not sure of what we need, is a hard problem. 
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Figure 1. Study design. 10 groups, their Self and Group 
lists about each participant, the three types of similarity 
ratings. There was also a survey and a set of interviews 
which captured context and sentiment. 



Expertise location, for this reason, has been a focus of 
the knowledge management field for many years. Knowl- 
edge management has also focused on the process of or- 
ganizational learning and dissemination of that learning 
within the organization. In many cases, this has been 
done through the tracking of created documents and 
other knowledge artifacts (Martin 2008). 

An additional approach should consist of uncovering that 
which has not yet been recorded - that information 
which is in the heads of a group's membership. We 
should be equipped to hold up a mirror to help reflect an 
organization's insights and expertise back on itself. We 
need to help uncover the dark corners where we are not 
sure about the expertise in the room. With a regimen 
of self-reflection, iterated over time, I hope this problem 
can be made less hard. I think we can discover whom 
to ask for the relatively low cost of a little sustained 
individual effort and some focused record-keeping in the 
distributed network. 



STUDY DESIGN 

Because cognitive authority is a subjective measurement, 
there is no objective way to measure its "precision" , no 
yard stick by which to measure its correctness. How- 
ever, if, with repeated exposure and more familiarity, a 
group's assessment and the assessment of the individual 
they are assessing become more similar over time, then 
I argue that this is a signal of the collective assessment's 
relative validity. This validity is what makes the group's 
opinion matter. This validity is what gives a group's 
opinion its weight. 

This study had 10 different groups of individuals, mostly 
coworkers, use free text keywords to label each others' 
areas of expertise (Figure [l]) . The participating groups 
consisted of members from a family retail business, a 
dentist's office, two distributed software development 
groups, a museum education staff, a writer's network, a 
legal non-profit, a global engineering firm, an academic 
faculty group, and an academic administrative office. 



Results were shared back into the group and made vis- 
ible, and the process was repeated for up to five to- 
tal rounds. The resulting product was an aggregated, 
weighted list of words associated with each person's ar- 
eas of expertise. Inspired by the distribution of key- 
words affiliated with individual URLs at delicious.com, 
each individual's weighted list can be viewed as a specific 
fingerprint in the multidimensional space created by all 
possible keywords. 

While individuals were labeling each other during each 
round, they were also labeling their own areas of exper- 
tise. With each passing round, ala the Delphi model 
(Dalkey & Helmer 1963, Rohrbaugh 1979, Stewart 1987, 
Luo & Wildemuth 2009) , the participants were presented 
with what their group had said about them and had 
the opportunity to update (by adding, removing, or ab- 
staining) their list of keywords about their own areas of 
expertise. 

The original Delphi study was run in the 1950s and 1960s 
by the RAND corporation to help the US Government 
determine the nuclear capabilities of the Soviet Union 
(Helmer & Rescher 1959, Dalkey & Helmer 1963). They 
were studying the unknown military futures market by 
asking a variety of experts to answer a battery of ques- 
tions. The answers were collated and then distributed 
back to the experts for additional rounds of answering 
the same questions - but critically, with the collective 
opinions of the other experts to aid their synthesis. 

Rowe and Wright write that, "in particular, the struc- 
ture of the technique is intended to allow access to 
the positive attributes of interacting groups (knowledge 
from a variety of sources, creative synthesis, etc.), while 
pre-empting their negative aspects (attributable to so- 
cial, personal and political conflicts, etc.)" (Rowe & 
Wright 1999). Over the following four decades, the Del- 
phi method has been refined and used in many other 
areas besides military futures, including social science 
predictions (Linstone & Turoff 1975, Rowe, Wright & 
McColl 2005, Hsu & Sandford 2007). 

Most research has suggested that with proper prepa- 
ration and consideration for expert subjects, question- 
naires, and evaluation, a Delphi study can run from 
three to five rounds, with four being the most com- 
mon number of iterations (Hsu & Sandford 2007). Some 
prior Delphi studies have used post-task surveys to sam- 
ple participants' reactions - from satisfaction (Van De 
Van & Delbecq 1974) to confidence (Scheibe, Skutsch & 
Schofer 1975, Boje & Murnighan 1982) to difficulty and 
enjoyableness (Rohrbaugh 1979). I employed some of 
the same types of questions here, especially considering 
the subjects were being asked to formalize their informal 
knowledge about one another. 

A traditional Delphi study involves 1) an objective fa- 
cilitator who gives "controlled feedback" in the aggre- 
gate, 2) a collection of independent experts in a domain 
(anonymous, to each other), and 3) a series of evalua- 
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tions (iterations) designed to have the collective opinion 
of the experts predict the future in that particular do- 
main (Rowe & Wright 1999). 

In this study, the co-workers are the experts, their labels 
are handled by the software and not attributed to any 
individual labeler, and there are 5 rounds of reflection 
and labeling. 



Expertise Tagging 



2 taiwan 
2thailand 

[Show SinaleTaaa) 



The similarity of a group's opinion and an individual's 
opinion will increase over time. That is, a similarity 
rating comparing two groups of words will increase from 
round to round of labeling. 

Furthermore, the warranting principle suggests that we 
give more credence to information provided by others, 
rather than information within the control of a particu- 
lar other (Walther k, Parks 2002, Walther, Heide, Hamel 
& Shulman 2009). Online or offline, information that is 
known to be easily manipulated is less trusted. Addition- 
ally, Delphi-style studies increase the confidence levels of 
the participants (Rowe et al. 2005). This suggests the 
second hypothesis: 

Hypothesis 2. Group members will have confidence 
in this system and exhibit increased trust in one another. 

RESULTS 

Similarity ratings were generated in three distinct ways: 
Human coded (gold standard), Amazon's Mechanical 
Turk (for scale), and an Algorithmic (automated) so- 
lution based on a bag-of-words assumption. These three 
methods, along with some survey results, are described 
here. 

Trained Human Similarity (HumanSim) 

The HumanSim ratings were designed to serve as a san- 
ity check for this type of data collection and analysis. 
Six trained raters coded a sample of the entire dataset 
and made 2348 comparisons of sets of words and rated 
their similarity on a 1-7 Likert scale with 1 meaning very 
dissimilar and 7 meaning nearly identical. 

Trained Humans 
or 

Mechanical Turk Workers 



Figure 2. Sample results from the labeling exercise 
demonstrating the Self and Group lists. 



Self List 



Data collection for this study was done primarily through 
a custom web interface that guided the participants 
through three stages per round, for up to five rounds. 
The three stages included reflection on the existing la- 
bels in the system (see an example in Figure [2]), a chance 
to add, update, or delete labels about their own areas of 
expertise (Self), and a chance to do the same for each of 
their peers (Group). 

Regarding the inherent "fuzziness" of this type of data, 
Latour, Woolgar, and Nelson suggest to us that where 
there is a lack of contention, a social fact will be defined 
(Latour & Woolgar 1986, Nelson 1993). Social tagging 
phenomena have demonstrated a stabilization of tagging 
behavior over time (Russell 2006, Colder & Huberman 
2005). Together, these suggest the first hypothesis: 

Hypothesis 1. As the social fact of what a person 
knows is collectively molded by the group, a consensus 
will appear and converge. 



V Similarity 

HumanSim S {1,2,3,4,5,6,7} 
TurkSim G {1,2,3,4,5,6,7} 



Group List 



Figure 3. Similarity Rating Flow: For both the Human- 
Sim and TurkSim (next section) ratings, a Self list and a 
Group list were compared against one another and rated 
on a 1-7 scale with a high rating of 7. 

They compared both weighted and unweighted lists and 
evaluated both random pairings (a Self list from one per- 
son with a Group evaluation of a different person) and 
lists "belonging" to a study participant (where the Self 
and Group lists were about that participant). Each com- 
parison was evaluated in an average of 15 seconds. 
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Figure 4. HumanSim: Trained humans determined that 
random pairings of Self and Group were dissimilar and 
that the Self and Group lists for the same individual were 
similar. When the labels were weighted, the similarity 
ratings were more tightly clustered at the high end of the 
scale. 



A two-way analysis of variance (see Figure El) illustrates 
the significance of both independent variables. There 
were no significant interaction effects. The main effect 
between the random design and study design was sig- 
nificant with a p- value of 0.000. The main effect for 
weightedness was significant at the 0.01 level with a p- 
value of 0.0013. 

Trained humans determined that labels attributed to 
one's areas of expertise by one's peers are similar to 
the labels given by someone about their own areas of 
expertise. The next two methods of generating ratings 
are an attempt to do the same thing, but cheaper and 
more quickly. 

Untrained Mechanical Turk Similarity (TurkSim) 

The second of the three methods of calculating similar- 
ity of the study data was via Amazon's Mechanical Turk. 
This method of analysis evaluated a series of 8773 Hu- 
man Intelligence Tasks (HITs) for a total cost of $219.33. 

Similar to the HumanSim comparisons, the Turkers were 
asked to rate the similarity of two sets of words on a 1-7 
Likcrt scale. A rating of 7 meant they felt the two lists 
were extremely similar and conveyed the same informa- 
tion. The Turkers rated data from each member of each 
group from each of the rounds. 

When looking at the similarity ratings across the en- 
tire study (see Figure [5]), it is clear the agreement that 
the lists convey similar information, but there is not a 
dramatic rise in that agreement over time. The range 
of similarity scores tightened around the mean as the 
rounds progressed. This does show consensus-building, 
but not an increase in the raw similarity of the data 
being evaluated. 

Untrained Mechanical Turk workers determined that la- 
bels attributed to one's areas of expertise by one's peers 
are similar to the labels given by someone about their 
own areas of expertise. However, that similarity did not 
measurably increase over time. 



Figure 5. TurkSim: Untrained Mechanical Turk workers 
evaluated the two sets of labels about study participants' 
areas of expertise to be similar (average rating of 5) with 
the range tightening noticeably as the rounds continued. 
The ranges collapsed more quickly (Round 2 vs. Round 
5) when all the words were shown, rather than only the 
words that appeared in the lists multiple times ("com- 
mon" words). 



Algorithmic Similarity (AlgSim) 

The third method for evaluating similarity of the study 
data was automated and tested here to see whether it 
produced reproducible results, similar to that of our Hu- 
manSim gold standard and TurkSim methods. 
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Figure 6. Similarity Rating Flow: For the AlgSim rat- 
ings, both the Self list and the Group list were "sense dis- 
ambiguated" by using a package of the WordNet project 
which determines which version of each word is being used 
(i.e. bank: money vs river). Then each list is compared 
as a group against the other and rated by the algorithm 
in Equation [T] 

The ratings in this section were computationally gen- 
erated based on an algorithm (Equation [T]) originally 
defined to calculate the similarity of sentences in the En- 
glish language (Mihalcea, Corley & Strapparava 2006). 
It uses a naive bag-of-words approach and ignores con- 
text such as word order. This naivety is a positive for this 
analysis since there is no implicit word order in the sets 
of labels being evaluated. However, the labels are related 
in that they are part of the same set and this algorithm 
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allows a straightforward approach to generating a value 
for that relationship. The resulting similarity scores were 
in the range [0..1] and cannot be directly compared to the 
human-generated Likert scale scores of 1-7 from the prior 
two sections. The original lists of labels were sense dis- 
ambiguated using the WordNet database (Pedersen, Pat- 
wardhan & Michelizzi 2004, Pedersen & Kolhatkar 2009) 
and then compared against one another. 



MgSim{A, B) 



{maxSim(w, B) * idf(w)) 

E 



E id -fM 



»//'(«')) 



(1) 



The algorithm took each word in set A and found the 
most similar word in set B (represented by maxSim(w, B)) 
and then multiplied by the information content of that 
word (represented by idf(w)). This summation was nor- 
malized across the information content of the entire list 
(E«je{ J 4} idf(w)). After each list was compared one to 
the other, the similarity values were averaged for the 
final AlgSim value for two lists A and B. 

Using this approach, the same analysis that was per- 
formed by the Mechanical Turk workers could be dupli- 
cated much more efficiently. The results are presented 
here. 
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Figure 7. AlgSim: The automatic algorithm detected a 
greater similarity in the later rounds than in the earlier 
rounds. It produced a tighter range of scores when it had 
more information to use (when using all the words rather 
than just the words used more than once in each list). 

Using this method the scores appear low on an absolute 
scale, but should be interpreted relatively. There was a 
clear rise in similarity score from Round 1 to Round 5 
(see Figure [7]) for both evaluation techniques (using all 
the words and only the words which occurred more than 
once) . 



Table 1. AlgSim: Repeated measures ANOVA by Round 



Df Sum Sq Mean Sq F value Pr(>F) 
Round 1 0.32 0.32 10.22 0.0015 

Residuals 286 8.85 0.03 



The repeated measures analysis of variance (Table [1) of 
the AlgSim scores across time (represented by Round) 



for the case where all words were evaluated showed a 
statistically significant main effect with an alpha of 0.01 
and p- value of 0.0015. 



Table 2. AlgSim: Post-hoc ANOVA p-values by Round 





Round 1 


Round 2 


Round 3 


Round 4 


Round 2 


0.0249* 








Round 3 


0.0058* 


0.6428 






Round 4 


0.0016** 


0.3906 


0.6783 


- 


Round 5 


0.0014** 


0.3480 


0.6062 


0.9072 



However, looking at round by round post-hoc ANOVA 
analysis when all the words are used (Table [2]) , the only 
significant differences appear to occur after the original 
feedback loop between Rounds 1 and 2 (p-values less 
than 0.05) when the participants are initially faced with 
their group members' feedback and labels. No other sin- 
gle round or span of rounds show an effect. This same 
analysis did not produce significant effects when only the 
words appearing more than once were used. 

Survey 

The survey accompanying this study consisted of 54 
questions and was completed by 56 of the 64 participants 
across all 10 groups. 

The survey results listed in Table [3] represent the 11 orig- 
inal questions as well as the aggregated results from the 
seven included scales (see Table [3]). 

The scores are on a 1-7 Likert scale representing agree- 
ment: l=Extremely Disagree, 4=Neutral, 7=Extremely 
Agree. 

All but one of the original items scored with mild to 
strong agreement. The highest ratings of agreement were 
received by the statements regarding comfort and famil- 
iarity of the group members with one another's areas of 
expertise. Additionally, nearly all participants rated this 
to be an interesting exercise. 

Slightly lower ratings were received by the items regard- 
ing the results of the exercise. The participants believed 
the system gave them somewhat good and new infor- 
mation that they found useful. They also thought the 
system did not necessarily gather all the important ar- 
eas of their expertise and that they would not necessarily 
use the information to help them make decisions moving 
forward. 

Being averages, the aggregate scales are relatively mild 
and all fit between 3 and 5, straddling the Neutral rat- 
ing. However, they showed similar results to the original 
items. 

At the top of the list, the participants believed this exer- 
cise provided good data quality and was easy to use and 
clear to understand. The participants rated the items 
regarding the results of the exercise and its fit within 
their organization slightly higher than neutral. 
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Table 3. Survey: Items from Selected Scales 
Result Demonstrability (Moore & Benbasat. 1991) 

- I would have no difficulty telling others about the results of using this 
system. 

- I believe I could communicate to others the consequences of using this 
system. 

- The results of using this system are apparent to me. 

- I would have difficulty explaining why using this system may or may not 
be beneficial, (reverse coded) 

Relative Advantage (Moore k Benbasat 1991) 



Table 4. Survey Scales and Ratings 



■ Using this system would enable me to accomplish tasks more quickly. 



Using this system would improve the quality of work I do. 



- Using this system would make it easier to do my job. 

- Using this system would enhance my effectiveness on the job. 

- Using this system would give me greater control over my work. 



Performance Expectancy (Vcnkatesh, Morris, Davis & Davis 2003) 
- I would find this system useful in my job. 



- Using this system enables me to accomplish tasks more quickly. 

- Using this system increases my productivity. 
Effort Expectancy (Venkatesh et al. 2003} 



My interaction with this system would be clear and understandable. 



- It would be easy for me to become skillful at using this system. 



- I would find this system easy to use. 

- Learning to operate this system would be easy for me. 
Facilitating Conditions (Vcnkatesh et al. 2003) 



■ I have the resources necessary to use this system. 



I have the knowledge necessary to use this system. 

- This system is not compatible with other systems I use. (reverse coded) 
Anxiety (Venkatesh et al. 2003) 

- I feel apprehensive about; using this system. 



- It scares me to think that I could 1< 
system by hitting the wrong key. 

- 1 hesitate to use this system for fear of makin; 



lot of information using this 
mistakes I cannot correct. 



■ This system is somewhat intimidating to me. 



Data Quality (AVang tV Strong 1990) 



- This system produced data in conformance with the actual or true values. 
■ This system produced data that is applicable and relevant to my job. 

- This system produced data that is intelligible and clear. 



- This system produced data that is easily accessible. 



DISCUSSION 
HI 

The data supports the view that this method provides 
a baseline for concluding that a group's opinion about 
a person's areas of expertise can give good information. 
A consensus appeared, was agreed to by the individual 
being labeled, and somewhat converged over time as the 
language and norms of the group were negotiated in a 
shared space. 

This finding comes with the caveat that the participants 
knew one another well enough or had enough experience 
with one another to feel the data being provided was of 
good enough quality. When conducted outside of well- 
known groups, this finding may not hold as both partic- 
ipant identity and the promise of future interactions are 
not as strong. 

H2 

This hypothesis was found to be partially supported. 
Participants did have confidence in the system to col- 
lect and then report the type of information they were 
expecting it to report. They thought the data would be 
quality data and they trusted it for what it was. 

However, they did not report that the trust in the data 
carried over to increased trust in the other participants. 
The study design forced the group members to already 



Original Items 



Average 
Rating 



I am comfortable with my group's tags 5.439 
about my areas of expertise. 

I am happy with my group's tags about 5.351 
my areas of expertise. 

I am familiar with my group members' ar- 5.333 
eas of expertise. 

This was an interesting exercise. 5.196 

My group members are familiar with my 5.175 
areas of expertise. 

My group did not list important areas of 4.764 
my expertise. 

I am confident that this system gives me 4.696 
new information. 

This was a useful exercise. 4.679 

I am confident that this system gives me 4.643 
good information. 

I am willing to incorporate output from 4.607 
this system into my decision making. 

I would be more comfortable with my 3.298 
group's tags if the tags were not anony- 
mous. 



Scale 



Average 
Rating 



Data Quality 


4.709 


Effort Expectancy 


4.670 


Result Demonstrability 


4.299 


Facilitating Conditions 


4.250 


Performance Expectancy 


3.836 


Relative Advantage 


3.742 


Anxiety (reverse coded) 


3.036 



be acquainted with one another and have existing work- 
ing relationships. This means that the participants be- 
gan the study with a fairly high degree of trust. This 
study provided no support for the idea that participants' 
trust levels increased because of the exercise. 

It would be interesting to ask a specific set of questions 
about colleague trust of a set of group members who were 
just beginning to work together or of group members who 
knew each other in a less formal environment than their 
salaried jobs. 

CONCLUSION 

Throughout this work, there was a sense that a kind of 
signaling was happening among members of the group, 
but implicitly through the data. The participants did 
not report they brought the ongoing results of this study 
into their face-to-face conversations. 

In a social space, judgment from others is ever-present 
and constant. By bringing that sense of evaluation out in 
the open and working together, a new collective artifact 
was produced that changed over time. This artifact held 
importance in the minds of the participants as they were 
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looking squarely at the group's judgment of their skillsets 
and areas of expertise. Their value to the group was laid 
bare in some sense, made explicit. 

The participants reported the first round was the hardest 
- tough on the psyche - but also exciting, and potentially 
rewarding. They also reported ongoing concern about 
being pigeon-holed for what they have already done or 
already shared. Perhaps this type of public judgment 
encourages exploration and continued learning on the 
behalf of the individual. Perhaps it discourages sharing 
in the longer term. 

Overall, this research has provided insight into how fa- 
miliar groups of individuals in the workplace can under- 
stand what their colleagues think of their areas of exper- 
tise. This work has shown that, with simple keywords, 
group members can convey the salient areas of expertise 
of their colleagues to a degree that is deemed "similar" 
and of "high quality" by both third parties and those 
being evaluated. 

Identity formation and negotiation is alive and well, and 
this research fits within the frames drawn by (Goffman 
1959) and (Tajfel & Turner 1986) and furthered by 
(boyd 2002, boyd 2008). We perform and we under- 
stand ourselves in part by understanding the reflections 
that come back to us from others (Marchionini 2009). 

In a fast-moving networked workplace, this ability to 
gain insight into the knowledge of others with a simple 
trustable lookup may prove valuable. Tapping into the 
collective understanding and distilled opinion of those 
around us could be a useful tool or sanity check against 
both direct and indirect individual claims of expertise. 
Equally, it could serve as a weapon against misplaced 
modesty, allowing us to collectively reward those who 
deserve to be given credit when credit is due. 

What remains an open question is whether this type of 
collective opinion mapping works in an environment be- 
yond the walls of the relatively small, trusted workplace, 
where people know one another (stable identity) and 
have many incentives to behave and only say positive, 
professional things about one another ("the shadow of 
the future" (Axclrod 1984)). 

I hope to begin answering this larger question soon with 
work based on the open internet. 

We now live in an ever-shrinking world of always-on con- 
nectivity and powerful communication devices. Since 
these devices are two-way, they provide a voice (and a 
distribution platform) to millions who, prior, have never 
had a voice. This is a remarkable achievement and serves 
as a testament to the incredible advance of technology 
and our collective striving for equality with regards to 
opinions and freedom of speech. However, with monu- 
mental increases in the number of voices and opinions be- 
ing shared, we demand a requisite increase in the power 
of tools to help us filter all this newfound information. 
We need good knobs to help us determine where to direct 



our always-limited and increasingly precious amount of 
attention. 

The freedom to listen to anyone has to be balanced with 
the practicality of not being able to listen to everyone. 
We need tools that help us serve both of these needs, 
albeit not at the same time. The tools need to be flex- 
ible enough to let us listen to whomever, whenever and 
wherever we want, and to reserve the right to change our 
minds at a later time. 

Finding good sources of information is hard. Knowing 
whom to listen to when the subject matter is beyond 
one's personal experience is a daunting and important 
problem, but one that can be reduced to an engineering 
problem with the right approach. 
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